Rethinking Bi-Level Optimization in Neural Architecture Search: A Gibbs Sampling Perspective

One-Shot architecture search, which aims to explore all possible operations jointly based on a single model, has been an active direction of Neural Architecture Search (NAS). As well-known one-shot solution, Differentiable (DARTS) performs continuous relaxation the architecture's importance and results in bi-level optimization problem. However, as many recent studies have shown, DARTS cannot always work robustly for new tasks, is mainly due approximate solution optimization. In this paper, neural search addressed by adopting directed probabilistic graphical model represent joint probability distribution over data model. Then, architectures are searched optimized Gibbs sampling. We rethink problem task sampling from posterior distribution, expresses preferences different models given observed dataset. evaluate our proposed NAS method -- GibbsNAS space used DARTS/ENAS NAS-Bench-201. Experimental multiple show efficacy stability approach.